Systems and methods for calculating text difficulty

ABSTRACT

Disclosed are systems, methods, and products for language learning that automatically extracts keywords from resources using various natural-language processing product features, which can be combined with custom-designed learning activities to offer a needs-based, adaptive learning methodology. The system may receive resources having text and then determine a text difficulty score that predicts how difficult the resource is for language learners based on any number of factors, including any number of semantic and syntactic features of the text. Training resources labeled with metadata may be used to train a statistical model for determining difficulty scores of newly received text. Resources may be grouped based on difficulty score, and groups of resources may correspond to language learners&#39; proficiency levels.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/765,105, filed Feb. 15, 2013, which is incorporated byreference in its entirety.

FIELD OF THE DISCLOSURE

The subject matter disclosed herein relates generally tocomputer-assisted language learning.

BACKGROUND

Conventional language learning methodologies organize learning materialinto lessons, which may often contain metalinguistic instructionalinformation followed by educationally-oriented exercises. Having pupilsdemonstrate their knowledge of particular subject matter in variouseducational activities comprising some series of questions is known.Also known are educational activities in which pupils demonstrate theirknowledge by engaging in various tasks. In many cases, educationalsoftware may implement educational activities in which pupilsdemonstrate their knowledge through a series of questions or by engagingin various tasks.

Learning activities are ordinarily prepared manually by a teacher,textbook author, or other curriculum planner. Learning activities arecommonly prepared by an instructor, educational expert, or some otherparty who is preparing educational content and who is also familiar withaspects of language learning. Typically, once these learning activitiesare generated they are then reproduced en masse. Similarly, conventionaldistance learning programs rely on prepackaged language learningsoftware or traditional textbooks. In such prepackaged software andtextbooks, curriculum planners create content and then try to have thesame content serve many people. A problem with this paradigm is thatlearning activities are not dynamically generated to suit the needs of alanguage learner.

Conventional language-learning software and tools employ a teachingmethodology, curriculum, and coursework that will remain static unless areplacement or supplemental textbook or language learning product ispurchased by a language learner or developed by an instructor. Not onlyis a curriculum not adaptable but, with each page or new computerassignment, there is no adaptability of activities and learningmaterials on a more granular scale. The overall curriculum is static, sodaily activities and skills exercises cannot adapt to address thelearner's needs. For example, a learner's past tense verb conjugationmay not be a problem, but the learner's spelling may be weak.

Conventional language methodologies often implement distractors, whichare pre-determined incorrect answers to academic questions, such asmultiple-choice problems. However, conventional language learningmethodologies implement distractors that are entirely static and cannotadaptably address learners' weaknesses of particular skills. Moreover,known language learning tools are not adaptable to suit a learner'sgoals or content preferences. Learners are left to work with the staticlanguage learning materials supplied to them, regardless of learners'personal interests or the real-life applicability of the those materialsin the context of the learners' goals.

What is needed is an efficient and effective way to dynamically createcontent and learning materials, or learning activities, for a languagelearning course. What is also needed is a way to dynamically generatelearning activities and language learning coursework that may even adaptautomatically mid-course, to suit a language learner's needs, goals, andinterests.

As discussed above, developing distractors for the purposes of languagelearning is known in the art. However, distractors are commonly preparedin some wrote or manual manner so that they may be reproduced in volume.In many cases, when developing learning activities based upon a certaintext, the activities must focus on some set of words to exercise variouslearning objectives. Usually, a person preparing learning activitiesmanually identifies and develops the words upon which learningactivities will be based or focused.

What is needed is a way, in a language learning context, toautomatically identify words in text, which may be useful to languagelearning, and then extract those words from text. What is needed is away to extract useful words of text, or keywords, and then store thosekeywords for use in developing learning activities.

Identifying keywords in a text to generate a summary of the content ofthe text is known. Conventional keyword extractors are typicallyinterested in obtaining a percentage of keywords to provide a summary ofthe text as efficiently as possible. For efficiency, conventionalkeyword extraction tools seek to summarize the content of the text usingas few keywords as possible. But, such conventional keyword extractiontools may incidentally filter out keywords that would otherwise behelpful for language learning, rendering conventional keyword extractiontools ineffective to language learning contexts.

What is needed for language learning is a way to obtain many keywordsfrom text, regardless of whether keywords appear inefficient insummarization contexts. What is also needed is a way to extract andstore keywords that are useful for language learning.

There are known means for identifying various attributes of a word intext, e.g., noun, past-tense, first-person, etc. It is also known tofilter out various inconsequential words from a line of text. Forexample, it is known to remove articles such as “the” or “a” from anonline search query through the use of a stop word list. Conversely,words which are rare in a corpus, having a lower probability ofoccurring in a given document, can be given weight in an informationretrieval system through known techniques such as TF-IDF.

What is needed is a means for identifying and quantifying thepedagogical value of keywords extracted from text for language learning.What is needed is a way to look at a word's attributes to determinewhether the word may help a pupil learn a language.

There are known methods in the art for calculating text difficulty.Conventional methods for calculating the difficulty of a corpus aredirected toward adult native language speakers. Conventionalmethodologies of calculating text difficulty is usually doneholistically by teachers and textbook writers. But, there is noestablished method of calculating a text difficulty that is tailored forsecond language learners. For native-language readers, and for childrenlearning to read in their first language, there are accepted benchmarksdetermining text difficulty. There are known systems of qualifying bookson levels, such as leveled readers from A-to-Z, something that mostschoolchildren learn while learning to read.

Moreover, there are many known readability scales, such asFlesch-Kincaid, which measures the number of words in a sentence and thenumber of words total. But, known text difficulty calculation methodsare designed for learners who already speak the language that they arethen learning to read. When teaching a second language to non-nativespeakers, particularly young adults and adults—many of whom already knowhow to read in their native language, but simply do not know how to readin the second language—there are different challenges that make readingtexts in the second language difficult for them when compared to theschoolchildren learning their native language.

Known first-language text difficulty levels and calculations are notalways applicable in situations of adult second-language learners, ornon-native speakers. First-language learners are learning to read as askill, and learning to read is something that takes years, whereassecond-language learners typically know how to read in their nativelanguage.

What is needed is a means of learning a language that applies a meansdifferent from known means of learning a native language. Typicallyschoolchildren are learning that the word “ball” maps to the conceptthat they already have for the entity, a ball. Adult language learners,on the other hand, already know how the words may map to certainconcepts. So rather than having them start with “that is a ball,” adultlanguage learners could start with something inherently morecomplicated, like “that is an electron,” or “I'll take a half-caff lattewith organic soy milk.” Adult second-language learners typically knowhow to read complicated items in their own languages. It is desirablefor the adult learner to understand these items in a second-language.

Known text difficulty calculation methods may measure text difficulty incomparison to native speakers through their development from childhood.For second-language learners the scale should be different. For example,a non-native speaker may review a long block of text containing verycomplicated ideas in the second-language, but he or she will have noproblem understanding the concepts. This is particularly true if thetext contains complicated ideas but is written in a way that exercisessimplified language. The ideas are no less complicated. On the otherhand, a simple concept like the weather forecast may incomprehensiblefor a novice language learner if it is written in a linguisticallychallenging way. This is because the issues second-language learnershave are not the same as first-language learners.

What is needed is a way to calculate a text difficulty of a corpus fornon-native speakers. What is needed is a way to calculate a textdifficulty of a corpus that is more suited for adult language learners.Text difficulty must be calculated primarily according to theidiosyncrasies of the language that make learning that languagedifficult. What is also needed is a way to automatically determine theappropriateness of text based on the calculated difficulty to thenautomatically prepare language learning coursework.

Generating distractors for educational exercises is known in the art.Distractors are ordinarily written manually by humans to prepare forvarious forms of language learning materials, such as a textbook orproblems that exercise skills. Afterwards, the attendant languagelearning materials can the be reproduced en masse. Usually, a teacher orother curricula expert writes questions about the form or content of aresource and then comes up with answers. This means that the teachingmethodology, curriculum, and coursework will remain static unless adifferent textbook or language learning product is purchased. Thismanual effort is also inefficient and costly.

What is needed is a way to dynamically generate distractors for languageleaning. What is needed is a way to adapt distraction generation andselection based on learner's needs. What is needed is a way toautomatically generate distractors from a resource. What is also neededis a way to prepare distractors that may adapt to the various types ofresources (e.g., a document containing text, an audio playback, a videoplayback) used for preparing learning exercises.

SUMMARY

The systems and methods described herein include variousnatural-language processing product features that can be combined withcustom-designed learning activities to offer a needs-based, adaptivelearning methodology. The system may receive a resource, extractkeywords relevant to non-native language learners, assign a difficultyscore to the resource, generate a definition as well as a list ofpotential distractors for each keyword, and topically tag the resourceagainst a taxonomy based on the content. This output is then used inconjunction with a series of learning activity-types designed to meetlearners' language skill needs to create dynamic, adaptive activitiesfor them. Various components of the systems and methods are described infurther detail below.

In one embodiment, a computer-implemented method of language learningcomprises selecting, by a computer, from a resource store a resourcehaving content, wherein the selected resource is related to a contentinterest of a learner and has a resource difficulty level based upon aproficiency level of the learner; identifying, by the computer, in auser data store that stores one or more abilities of the learner forspecific language skills; identifying, by the computer, a specificlanguage skill for improvement based upon the one or more abilities ofthe learner; identifying, by the computer, a type of learning activitythat exercises the identified language skill for improvement;identifying, by the computer, a set of one or more distractors of a typesuited to the identified type of learning activity and having adistractor difficulty level based upon the ability of the learner in thespecific language skill; generating, by the computer, a learningactivity of the identified type of activity utilizing at least one ofthe set of one or more distractors; updating, by the computer, theability for the specific language skill in the user data store accordingto a result from the learning activity; and updating, by the computer,the proficiency level of the learner in the user data store.

In another embodiment, a computer-implemented method for facilitatinglanguage learning comprises identifying, by a computer, a particularlanguage skill to exercise; selecting, by the computer, a learningactivity that exercises the identified language skill; selecting, by thecomputer, a set of one or more distractors each of a type of distractorsuited to the selected learning activity, wherein each of the one ormore distractors are associated with a resource; and generating, by thecomputer, the selected learning activity, wherein the selected learningactivity comprises at least one distractor from the selected set of oneor more distractors

In another embodiment, a computer-implemented method for crafting alanguage learning pedagogy comprises: identifying, by a computer, a setof one or more learner capabilities of a learner comprising a languageproficiency level and one or more abilities at specific language skills;identifying, by the computer, a set of one or more learner preferencesassociated with the learner comprising one or more learner contentinterests; determining, by the computer, a type of learning activity forexercising a language skill based on an ability of the learner at thelanguage skill; determining, by the computer, a set of one or moredistractors associated with a resource to implement in the learningactivity according to the determined type of activity, wherein adistractor difficulty comparable to the one or more capabilities of thelearner; generating, by the computer, the learning activity of theactivity type, wherein the learning activity comprises the set ofdistractors; and generating, by the computer, a lesson comprising a setof one or more learning activities according to a learner goal in thelearner preferences.

In another embodiment, a computer-implemented method for tailoringlanguage-learning to a learner comprises: determining, by a computer, anability of a learner in a specific language skill based on a result of askill assessment performed by the learner; determining, by the computer,a language proficiency level of the learner based on at least onelanguage skill ability of the learner; receiving, by the computer, alearner content interest and a learner goal from the computing device ofthe learner; and storing, by the computer, a learner profile associatedwith the learner in a user data store, wherein the learner profilecomprises one or more language skill abilities of the learner, thelanguage proficiency level, the learner content interest, and thelearner goal.

In another embodiment, a language learning system for automaticallygenerating activities comprises: a host computer comprising a processorexecuting a set of software modules for language learning; a keywordstore storing a set of keywords extracted from a resource by a keywordextractor module in the set of software modules executed by the hostcomputer; a user data store storing data associated with a learner in alearner profile, wherein the learner profile comprises a set of learnerlanguage abilities and a set of learner preferences; and a learnercomputer comprising a processor configured to execute a user interfaceto interact with a set of learning activities generated by a learningactivity generator module executed by the host computer, wherein alearning activity is automatically generated using the capabilities andpreferences of the learner stored in the learner profile.

In one embodiment, a computer-implemented method for extracting keywordsfrom text comprises: parsing, by a computer, a set of one or morepotential keywords from text of a resource containing text; storing, bythe computer, into a keyword store each potential keyword in the setmatching a term in a computer file containing a keyword whitelist andeach potential keyword in the set matching a collocation in the keywordwhitelist; determining, by the computer, for one or more potentialkeywords in the set of potential keywords, a word difficulty valueassociated with each of the one or more potential keywords based onscoring rules determining the word difficulty value; and storing, by thecomputer, into the keyword store each potential keyword having adetermined pedagogical value that satisfies a threshold word difficultyvalue.

In another embodiment, a system comprising a processor andnon-transitory machine-readable storage containing a keyword extractormodule instructing the processor to execute the steps of: parsing textfrom a resource into a set of one or more potential keywords;identifying one or more collocations in the set of potential keywordsmatching a collocation in a file containing a keyword whitelist;determining a pedagogical value for each extracted word according to oneor more scoring rules; and storing a set of one or more extractedkeywords into a keyword store, wherein the set of extracted keywordscomprises each potential keyword having a word difficulty valuesatisfying a threshold value and each identified collocation.

In one embodiment, a computer-implemented method for predicting a textdifficulty score for a new resource comprises extracting, by a computer,one or more linguistic features having a weighted value from a pluralityof training resources containing text, wherein the text is associatedwith a metadata label containing a text difficulty score of the text;determining, by the computer, a vector value associated with eachtraining resource based on each of the weighted values of each of theextracted one or more linguistic features; training, by the computer, astatistical model using the vector values associated with each trainingresource, wherein the statistical model represents a correlation betweena set of features selected for extraction, a set of weighted valuesassigned to the set of features selected for extraction, and a set oftext difficulty scores associated with the training resources;extracting, by the computer, one or more linguistic features having aweighted value from a new resource; determining, by the computer, avector value for the new resource based upon the set of extractedlinguistic features; and predicting, by the computer, a text difficultyscore for the new resource based upon the vector value for the newresource and the statistical model.

In another embodiment, a computer-implemented method for determiningtext difficulty for a resource comprises: comparing, by a computer, atleast a portion of text related to a resource against a lexical featurebank file comprising a listing of semantic features and a listing ofsyntactic features; identifying, by the computer, a set of lexicalfeatures associated with the text based on the comparison, wherein theset of lexical features comprises at least one semantic feature from thetext matching a semantic feature in the list and at least one syntacticfeature from the text matching a syntactic feature in the list;assigning, by the computer, a first value to each of the lexicalfeatures in the set of lexical features; and determining, by thecomputer, a text difficulty score for the text using each of the firstvalues associated with each of the features in the set of lexicalfeatures.

In another embodiment, a system comprising a processor andnon-transitory machine-readable storage containing a text difficultycalculator module instructing the processor to execute the steps of:comparing text in a resource against a listing of features comprising alist of semantic features and a list of syntactic features; identifyingone or more semantic features in the text matching the listing offeatures, and identifying one or more syntactic features in the textmatching the listing of features; assigning a value to each of theidentified semantic and syntactic features in the text according to astatistical model, wherein the statistical model identifies the valueassociated with each feature in the listing of features; and determininga text difficulty score associated with the resource using each of thevalues assigned to the identified semantic and syntactic features.

In one embodiment, a computer-implemented method for facilitatinglanguage learning comprises: generating, by a computer, a set of one ormore semantic distractors comprising one or more words having adefinition that is related to a target word; generating, by thecomputer, a set of orthographic distractors comprising one or more wordshaving an edit distance satisfying an edit distance amount setting,wherein the edit distance is a number changes to a word required to beidentical to the target word, and wherein the edit distance amountsetting determines the number of changes to the word; and generating, bythe computer, a set of phonetic distractors comprising one or morehomophones of the target word.

In another embodiment, a system comprising a processor andnon-transitory machine-readable storage containing a distractorgenerator module instructing the processor to execute the steps of:receiving one or more keywords extracted from a resource containingtext; identifying in one or more dictionary sources, one or moredistractors that are of one or more distractor types and are associatedwith a keyword; and generating a set of one or more identifiedorthographic distractors comprising a word having a predetermined editdistance relative to the keyword.

Additional features and advantages of an embodiment will be set forth inthe description which follows, and in part will be apparent from thedescription. The objectives and other advantages of the invention willbe realized and attained by the structure particularly pointed out inthe exemplary embodiments in the written description and claims hereofas well as the appended drawings. It is to be understood that both theforegoing general description and the following detailed description areexemplary and explanatory and are intended to provide furtherexplanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure can be better understood by referring to thefollowing figures. The components in the figures are not necessarily toscale, emphasis instead being placed upon illustrating the principles ofthe disclosure. In the figures, reference numerals designatecorresponding parts throughout the different views.

FIG. 1 shows an exemplary embodiment of a language-learning system.

FIG. 2 shows a flowchart for an exemplary method embodiment of keywordextractor module.

FIG. 2A shows a flowchart for an exemplary method embodiment of keywordextractor module.

FIG. 3 shows a flowchart for an exemplary embodiment of a method ofdetermining a text difficulty score for language of text.

FIG. 4 shows a flowchart of an exemplary embodiment of the methodexecuted by the distractor generator module.

FIG. 5 shows a screenshot of an exemplary embodiment of a graphical userinterface presented to a learner to begin a language-learning lesson.

FIG. 6 shows a screenshot of an exemplary embodiment of a graphical userinterface for a learner to engage a reading comprehension activity.

FIG. 7 shows a screenshot of an exemplary embodiment of a graphical userinterface for a learner to engage a vocabulary activity.

FIG. 8 shows a screenshot of an exemplary embodiment of a graphical userinterface for a learner to engage a spelling activity.

FIG. 9 shows an exemplary method embodiment of a language systemimplementing a learning module.

DETAILED DESCRIPTION

The present invention is here described in detail with reference toembodiments illustrated in the drawings, which form a part here. Otherembodiments may be used and/or other changes may be made withoutdeparting from the spirit or scope of the present disclosure. Theillustrative embodiments described in the detailed description are notmeant to be limiting of the subject matter presented here.

Embodiments of the system may automatically develop one or more portionsof a series of learning activities of various intended learningoutcomes, e.g., improved vocabulary, improved reading comprehension,improved overall learner proficiency. Learning activities may pull dataautomatically generated by various software modules that are executed onone or more computing devices, and/or data of various types stored in aresource store.

Modules generating data may be a keyword extractor, a distractorgenerator, and a text difficulty calculator. The various components ofthe language learning system may develop output used by a learningmodule as input for portions of lessons generated by the learningmodule. Thus, the learning module may dynamically create learningactivities suited for language learners based on a variety of resources.

In some embodiments, learning activities may use as input variousinformation describing aspects of the language learner such as alearner's interests, the learner's overall language proficiency level,and/or the learner's abilities in various particular language skills.For example, a resource used for a learning activity may be selectedbased on the learner's interest in sports. As another example, thelearning activity may be generated to exercise a particular languageskill where the learner needs to focus, which is determined according toan assessed ability in that particular skill (e.g., spelling, readingcomprehension). In another example, a more rigorous or well-rounded setof learning activities may be generated if the learner has a goal toprepare for the TOEFL as opposed to preparing for a vacation.

Resources selected to use in building learning activities may vary indifficulty, type, and content. For example, resources may belinguistically difficult or easy, relative to one another and/orrelative to learners' proficiency levels. As another example, resourcesmay be one or more portions of text of a document or transcript, audio,video, audiovisual, or images.

Reference will now be made to the exemplary embodiments illustrated inthe drawings, and specific language will be used here to describe thesame. It will nevertheless be understood that no limitation of the scopeof the invention is thereby intended. Alterations and furthermodifications of the inventive features illustrated here, and additionalapplications of the principles of the inventions as illustrated here,which would occur to one skilled in the relevant art and havingpossession of this disclosure, are considered within the scope of theinvention.

Language Learning System Components

FIG. 1 shows the components of an exemplary embodiment of alanguage-learning system 100.

The language learning system 100 of FIG. 1 comprises a language learnerserver 101, a distractor store 102, a resource store 103, a keywordstore 104, a user data store 105, a network 106, a content curatorcomputer 107, and language learner's computing device 108. As shown inthis exemplary embodiment, a language learner's computing device 108 maybe a language learner's computer 108 a or a language learner's smartphone device 108 b.

A language learner server 101 may be any computing device, such as apersonal computer, or any other computing device comprising a processorthat may be capable of executing one or more language learning modules.Embodiments of the language learning server 101 may comprise a keywordextractor module, a text difficulty calculator module, a distractorgenerator module, and/or a learning module.

In the exemplary embodiment the language learning system 100 in FIG. 1,the language learner server 101 is shown as a single device. However insome embodiments, the language learner server 101 may comprise multiplecomputing devices. In such distributed-computing embodiments, where alanguage learner server 101 comprises a plurality of computing devices,each of the computing devices may comprise a processor. And each ofthese processors may execute language learning modules that are hostedon any of the plurality of computing devices.

Embodiments of the language learning system 100 may comprise one or moredata stores 102, 103, 104, 105: a distractor store 102, a resource store103, a keyword store 104, and a user data store 105. Data stores 102,103, 104, 105 may be databases comprising non-transitorymachine-readable storage medium storing and retrieving data related toone or more modules executed by a device processor in the languagelearner server 101. Data stores 102, 103, 104, 105 may be a singledevice hosting the database, or data stores 102, 103, 104, 105 may bedistributed among a plurality of computing devices hosting databases.

In some embodiments of a language learning system 100, one or more ofthe distractor store 102, the resource store 103, the keyword store 104,and/or the user data store 105 may reside on a language learner server101. In some embodiments, such as the one shown in FIG. 1, the languagelearning server 101 is a single device and each of the data stores 102,103, 104, 105 are hosted on distinct devices, each communicating withthe language learner server 101. An ordinary artisan would appreciatethat data stores 102, 103, 104, 105 may communicate with the languagelearner server 101 over any number of communication means capable offacilitating networked communication between computing devices, e.g.,LAN, WAN, InfiniBand, 3G, 4G, or any other computing communicationmeans.

A distractor store 102 may be non-transitory machine-readable storagemedium storing one or more distractors that are related to keywords. Thestored distractors may be dynamically generated by a distractor moduleexecuted by a language learner server 101.

A resource store 103 may be a non-transitory machine-readable storagemedium storing one or more resources. Resources may be received from acontent curator device 107, a learner computing device 108, or anotherexternal data source, such as a website, blog, or news service. Aresource stored in the resource store may be one or more portions oftext of a document (e.g., book, article, webpage, journal), an audiooutput, a video output, an audiovisual output, an image, or acombination thereof. In some embodiments, a resource store 103 may alsostore metadata associated with stored resources. Non-limiting examplesof metadata may include information describing a text difficulty scorefor text in a document resource, the length of a resource, and/or thecontent contained in a resource.

In some embodiments of the resource store 103, resources may bemultimedia resources (e.g., audio, video, image). In such embodiments,the resource store 103 may store metadata related to a stored multimediaresource. For example, the textual transcript of an audio or audiovisualresource may be stored in the resource store 103 as associated metadata.As another example, metadata may contain one or more timestamps thatcorrespond to particular timing points in a video or audiovisualresource. Timestamps may be associated with a transcript or a otherdescriptive text of the resource, e.g., a keyword in view at particulartiming point, or a description of events. In another example, metadatamay contain descriptions of points (e.g., coordinates) for boundingboxes encompassing areas of an image resource. Some embodiments havemultimedia resources, metadata may identify one or more keywords in themultimedia resources. This metadata may correspond with keywords in akeyword store.

In some embodiments of the resource store 103, metadata related tomultimedia resources may be manually entered by a content curator usinga user interface of a content curator device 107. In some embodiments,resources may be automatically retrieved from a variety of externalsources or received from external sources on a regular basis. In theseembodiments, metadata associated with multimedia resources may beautomatically retrieved, received, or updated, from the external sourcepromulgating the associated resource. In some embodiments, the metadatamay be retrieved from various external sources storing data related tomultimedia resources already stored in the resource store 103.

A resource store 103 may store resources transmitted from a userinterface of a computing device 107, 108 within a language learningsystem 100 or from some external data source. The resource store 103 mayexecute search queries to search the resource store 103 for resources.These search queries may be received from language learning modulesexecuted by a language learning server 101, or from a computing device107, 108.

A keyword store 104 may store one or more keywords extracted fromresources. A keyword store 104 may store metadata associated with thekeywords, such as their originating resource from which they wereextracted, or a word difficulty score describing linguistic difficultyof the keyword. A keyword store 104 may receive and store keywords froma keyword extractor module that is executed on a language learner server101. A keyword store 104 may also store keywords input from a userinterface of computing device 107, 108. A keyword store 104 may executesearch queries to search for a keyword according to queries receivedfrom language learning modules executed by a language learning server101.

A keyword store 104 is a computer-readable storage medium that may storekeywords, keyword attributes, and/or various other means for retrievingdata stored in the keyword store 104, such as an offset or a uniquedatabase record key. In such embodiments of the keyword extractionmodule, the keyword extraction module proceeds to store certainpotential keywords into a keyword store 104, as keywords extracted froma corpus or resource. Various algorithms and/or other operating rulesinstructing the keyword extraction module may be used to determinedwhich of the keywords are to be stored into the keyword store 104.

A user data store 105 is non-transitory machine-readable storage mediumthat may store learner profiles containing information related tolanguage learners. A user data store 105 may receive and store data fromone or more language learner modules executed on a language learnerserver 101, or a user data store 105 may store data input from a userinterface of a computing device 107, 108. A user data store 105 mayexecute search queries to search for learner profiles according tocommands received from one or more language learning modules executed bya language learning server 101.

A network 106 may connect each of the computing hardware devices in thelanguage learning system 100 to one another. One having ordinary skillin the art would appreciate that there are many possible permutationsfor connecting the various devices and components in embodiments of alanguage learning system 100. Embodiments of a network 106 mayfacilitate hardware components of a language learning system 100 to beproximately located within a building, a campus, a municipality, oracross any geographic span.

As seen in the exemplary embodiment of a language learning system 100shown in FIG. 1, a network 106 may be a combination of public network,such as the Internet, and private networks. However, in someembodiments, the network 106 may only be a private network, or only apublic network. The network 106 of FIG. 1 represents a combination ofthe Internet and some internal private network. The network 106 may be amixture of various data communication technologies, i.e., LAN, WAN, and4G.

In some embodiments, a network 106 may only connect some of the hardwarecomponents of a language learning system 100 while some other hardwarecomponents are connected with the language learning system 100 usingdifferent technologies or a different network. For example, data stores102, 103, 104, 105 of FIG. 1 may communicate with the language learningserver 101 using, for example, InfiniBand, for proximately locateddevices. Moreover, in some embodiments one or more of the components mayreside on a single device.

An ordinary artisan would appreciate that a language learning system 100may be a distributed computing system using one or more networkingtechnologies and redundancies technologies that may employ a number ofknown underlying techniques for facilitating communication betweencomputing hardware components.

One having ordinary skill in the art would appreciate that thenetworking architecture shown in the embodiment of the language learningsystem in FIG. 1 does not in any way limit architectural permutationsfacilitating communications between various components of otherembodiments of a language learning system 100.

A content curator computing device 107 may be a computer, smart phone,server, a tablet, a gaming system, or other computing device comprisinga processor configured to implement a user interface to administer thevarious modules and components of a language learning system 100. Insome embodiments, the content curator computing device 107 may becapable of networked communications with various other computingdevices.

Embodiments of a content curating device may execute a user interfacethat may allow a content curator to review output from, and/or manuallyinput various pieces of information, resources, and/or metadata into themodules executed by the language learning server 101, or into datastores 102, 103, 104, 105 within the language learning system 100.

For example, in some embodiments, a content curator may store a resourcein a resource store and then manually enter curated metadata that isassociated with the resource, such as a timestamp for a video resourceor a transcript for an audio resource. The content curator computingdevice may attach metadata to resource for associating the resource witha keyword or identifying various other attributes of the resource. Insome embodiments, a content curator computing device may receiveinformation associated with a learner or input data into a user profileassociated with a learner. The information or data may be received froma content curator inputting the information or data on a user interfaceassociated with the content curator computing device.

In some embodiments, a content curator computing device 107 may allow acontent curator to act as a tutor in either live chat, telephonic,and/or video sessions with a learner's computing device 108, or incorrespondence, e.g., e-mail. It is to be appreciated that a contentcurator computer 107 may be a computing device networked to communicatewith a language learner server 101. However, a content curator computer107 may also be the same device as the language server 101.

A learner computing device 108 may be a computer 108 a, a smart phone108 b, a server, a tablet computing device, a gaming system, or anycomputing device having a processor configured to implement a userinterface to communicate with the various modules of the languagelearning server. It is to be appreciated that a learner computer 108 amay be networked to communicate with a language learner server 101, buta learner computer 108 a may also be the same device as the languageserver 101.

In some embodiments, a learner computing device 108 may engage into atutoring session with a content curator device 107 over a network 106.The tutoring session may be held over any video-calling technology,voice call technology, or instant messaging service, such as GoogleHangouts®, Skype®, VoIP, and/or instant messaging software. In someembodiments, this tutor session may not be a product of a third-partyvendor, and is instead native to the language learning system 100. Inother embodiments, the language learning system 100 may comprise a tutormessaging service for the tutor and the learner to message one anotheroutside of a tutoring session.

A learner computing device 108 may send and receive electronic messagingwithin the language learning system 100 or through conventionalmessaging services. A content curator computing device 108 may likewisefacilitate such communication tutoring between a content curator andlearner. A content curator device 108 may also include an interface forgenerating tutoring assignments, grading assignments, scoring learningactivities, and sending/receiving assignments with a learner. Theseassignments may be sent through the native messaging service or throughthe conventional messaging service.

In some embodiments, learners may take an initial proficiencyassessment. This assessment may be a test or quiz stored on the languageleaning server 101. The assessment may be presented to learners throughthe user interface of the learner's computing device 108. The assessmentmay be scored automatically by the language learning server 101, or maybe scored by a content curator using the computing device of the contentcurator 107. This assessment may provide an initial global learnerproficiency score and/or scores reflecting the learner's abilities inparticular language skills, which are stored in a user data profile. Theproficiency level and/or language skill abilities of the learner may beautomatically updated in the user data store 105. The modules of thelanguage learning server 101 may automatically update the user dataprofile in the user data store 105 based on the learner's performance inlearning activities, and in some embodiments, learners may take aperiodically recurring proficiency assessment through a learner'scomputing device 108 to level-up in proficiency level or in an abilityscore.

Keyword Extractor

When applied in a language learning context, a keyword extractor asdescribed herein may address shortcomings of conventional keywordextractors since conventional keyword extractors are not commonlyapplied to language learning. The goal of conventional keywordextractors is to save a human reader time by reducing the number ofwords they need to read to extract meaningful information from theresource. These algorithms are optimized for finding the smallest subsetof words which have a high information content, which results in thefiltering out of common words, phrases, idioms and other vocabularywhich lies in a more general topic domain. Embodiments of the keywordextractor described herein implement keyword extraction techniques forextracting keywords useful for language learning. Conventional keywordextraction techniques are not applied in context of language learningand as such, conventional keyword extraction can be ineffective.

Embodiments of a language learning system may include a keywordextractor, otherwise referred to as a keyword extraction module, whichis a component embodied on a computer-readable medium and executed by aprocessor. The keyword extractor may obtain one or more keywords fromtext. Keywords may be a particular word or set of words (e.g., phrase,collocation) recited in a resource.

Conventional keyword extraction techniques merely extract enough wordsto summarize the content conveyed by text. However, embodiments of akeyword extraction module described herein may be executed to gather asmany keywords as needed to facilitate language learning using aparticular document or set of documents. The keyword extractor mayimplement keyword extraction techniques adapted for language learning,which requires different information content identification andextraction heuristics. For example, a word which may not be closelyconnected to the meaning of a text would be discarded by conventionalkeyword extractors. However, this word may still have pedagogical valueto a language learner at a lower level of proficiency where generalvocabulary is being acquired.

As described herein, a keyword extraction module may, among otherfeatures, extract keywords from a resource that are pedagogicallyvaluable to language learning. Pedagogically valuable keywords may bewords in the text of a resource containing text, such as a document(e.g., article, journal, blog) or a transcript, which may aid languagelearning due to certain attributes or aspects associated with thosewords. As an example, a keyword may be pedagogically value when thekeyword is easily defined or when the keyword is central to the contextof the resource.

Embodiments of an extraction module may implement a keyword whitelist asa way of determining pedagogical value of potential keywords parsed fromtext. A whitelist may be any type of computer file containing wordspredetermined to be keywords to extract. In such embodiments, a keywordwhitelist may comprise words that are automatically stored into thekeyword store as an extracted keyword.

In some embodiments, a stop word file may be any computer filecontaining words that are never keywords for extraction. The stop wordfile may act as a filter against words that are not particularlyadvantageous to language learning.

In some embodiments, the keyword extractor module may implement one ormore scoring rules, which may be one or more algorithms for determiningpedagogical values of potential keywords. It is to be appreciated thatmultiple whitelists and stop word files may be used, or the whitelistand stop word file may be the same single computer file.

FIG. 2 is a flowchart for a method embodiment of keyword extraction fromthe text of a resource.

In step S200, after receiving a resource containing text, a keywordextraction module may begin the process of keyword extraction in step bygenerating a set of potential keywords. In some embodiments, step S200may begin with a step S201 in which the keyword extraction modulereceives a resource and then parses the text of that resources into aresulting set of potential keywords. Step S200 may optionally utilize astep S202 in which a keyword extraction module begins generating a setof potential keywords by preprocessing and/or standardizing the textparsed from the resource.

In a next step S204, embodiments a keyword extraction module may comparethe set of potential keywords against a keyword whitelist to determinewhether each potential keyword is pedagogically valuable.

Based on the comparison in step S204, in a next step S206, a keywordextraction module removes potential keywords, from a set of potentialkeywords, that match a filtered word listed in the stop word file.

Embodiments of a keyword extraction module may implement a stop wordfile that may filter out words pre-identified as lacking pedagogicalvalue. Non-limiting examples of stop words that are filtered out of theset of potential keywords may include: ordinal numbers, numbers, propernouns, and/or conjunctions.

In a next step S208, the keyword extractor module may store potentialkeywords matched to a word in the keyword whitelist file identified ashaving pedagogical value for language learning. Non-limiting examples ofsuch words may include collocations, verbal phrases, words or phrasesidentified as difficult for non-native speakers, words related to theresource, and/or words pre-identified as words critical to the languagebeing learned.

A keyword whitelist file may identify one or more collocations as havingpedagogical value, each of which may be kept as an extracted keyword.Collocations are words frequently found together in texts, such as “putout.” Language learners typically learn such collocations in chunks.But, conventional keyword extraction tools typically eliminatecollocations arising in a document. Embodiments of the keyword extractormay be adapted to specifically look for such collocations.

In a non-limiting example using “put out,” there may be circumstanceswhere the term “put” may otherwise be filtered out as lacking value, butphrasal verbs such as “put out,” “put up,” or “put on” might berecognized as collocations and, as such, they may be extracted by thekeyword extraction module. Language learners must learn each of theseusages of the word “put” because each of these instances of the term“put” have different meanings, and are equally distinct from the word“put.” In this example, the keyword extraction module facilitatesteaching a language learner each of these sets of collocations andvarious usages of the term “put.” Thus, language learners may see thedifference between “put on,” rather than only the term “put.”

Continuing with step S208, potential keywords identified by the keywordextraction module as having pedagogical value may be stored in a keywordstore. The keyword store is a computer-readable storage medium that maystore keywords, keyword attributes, and/or various other means forretrieving data stored in the keyword store, such as an offset or aunique database record key.

Embodiments of a keyword extractor module implement a keyword store maystore extracted keywords in the keyword store associated with keys ofn-grams, a set of one or more collocations, and one or more offsets. Asused herein, the term keyword may refer to a word or phrase comprisingone or more words, such as a collocation.

A keyword store stores keywords with keys of n-grams. An n-gram may be asequence of one or more words, n, recited within a unit of meaning in atext, which may refer to the length of the keyword. In other words, akeyword store may store unigrams, bigrams, and/or trigrams, or more,depending upon the length of the keywords being extracted.

For example, a unigram may be a word of text that stands on its own,thus “dog” is a unigram. Likewise, the phrase “dog walker” is a bigram,and the phrase “dog walking service” would be a trigram. Between each ofthe three examples of potentially extracted keywords, there is somegrammatical connection between three potential keywords. These threeexamples may be fitted together to form a unit of meaning, i.e., form aset of concepts together.

A keyword store may also store extracted keywords in association withcollocations, and/or offsets. Collocations in this context are similarto other database keys in that they may be stored in association with anextracted keyword in the keyword store, these collocation keys representa subset of one or more collocations reciting the extracted keyword in aresource, recited in an ongoing history of resources, or manuallyentered by a content curator. Offsets associated with the keyword in thekeyword store may be one or more values indicating the start and endpositions of the keyword associated with a resource.

After the keyword extraction module generates the set of potentialkeywords in step S200, the keyword extraction module may store all, or asubset, of the potential keywords, into the keyword store, as extractedkeywords. The keyword store may store only the keywords extracted from aresource. Some embodiments may allow this keyword store to build overtime, thereby accruing extracted keywords; some of these embodiments mayalso accrue the extracted keywords and associate them with theirrespective resources from which the keywords were extracted.

In a next step S210, a keyword extraction module may determine a scorefor each of the remaining potential keywords, i.e., potentials keywordsnot already eliminated or stored into the keyword store accordingscoring rules that evaluate each word's pedagogical value. Embodimentsmay score potential keywords using various permutations and combinationsof algorithms, software, and/or other tools, either known or disclosedherein, to determine the pedagogical value of a word.

As a non-limiting example, a keyword extractor may score bigrams andtrigrams using co-occurrence statistics, implemented by using a naturallanguage processing tool, such as the Natural Language Toolkit of thePython scripting language, to determine the likelihood that a wordshould remain a potential keyword. In this example, the keywordextractor determines the criticality of a word to the text by measuringthe word's frequency.

As another non-limiting example, a keyword extractor may refer to a listof collocations specifically configured for English learners at aparticular proficiency level.

As discussed later, as another non-limiting example, a keyword extractormay generate an additional word difficulty score to rank words in termsof likelihood of difficulty for non-native speakers to learn and master.Non-limiting examples of parameters used for calculating a word scoremay include a word's length and/or frequency of which the word'spart-of-speech is exercised in a broader population of external corpora,such as a term frequency-inverse document frequency (TF-IDF) score.

As shown in FIG. 2A, the keyword extractor may include an optional firstreview step S200 a and an optional second review step S218.

In a step S200 a, a keyword extractor implements a first review step ingenerating the set of potential keywords. Step S200 a may compriseoptional steps S201 a, S202 a, S202 b, and/or S202 c.

In a step S201 a, a keyword extractor may use known natural languageprocessing tools and techniques to generate the set of potentialkeywords.

In a step S202 a, the keyword extractor may implement the naturallanguage processing techniques to identify one or more attributesassociated with a potential keyword. In this step the keyword extractormay, for example, automatically identify a word's part-of-speech as anoun, adjective, verb, or otherwise. Other non-limiting examples ofattributes that a keyword extractor may automatically identify andassociate with potential keywords may include a topic and/or sub-topicfor a potential keyword, a number of syllables in a potential keyword, anumber of times a potential keyword appears in a resource comprising allof the resources stored in the resource store, one or more resourcesstored in the resource store, or other external resources or TF-IDF,and/or a definition for a potential keyword.

In a step S202 b, a keyword extractor may calculate a word difficultyscore for a potential keyword based on factors pulled from attributesassociated with the potential keyword. These factors may include, butare not limited to, TF-IDF of the word in the resource store, resourceor external sources, a number of syllables in a potential keyword, andwhether a potential keyword appears on an Academic Word List (AWL).

In a step S202 c, attributes of words may be identified and included inthe keyword store as metadata associated with a potential keyword. Forexample, metadata may include a part-of-speech, an originating resource,a definition of a word, or a context in which an extracted keyword isused in text. A context may be determined using metadata associated withthe resource in which a resource's content was identified and stored ina resource store or manually input. The keyword store may store apotential keyword in association with any resource content reciting thepotential keyword. Some embodiments of the keyword extractor mayimplement a tagging system for associating, or tagging, metadata withkeywords.

After step S202 c, a keyword extractor may perform one or more of thesteps between S204 through S216 shown in FIG. 2.

In an optional step S218, a keyword extractor may initiate an optionalsecond review of the text. In this step, each potential keyword and theassociated metadata may be presented to a user interface, of a clientcomputing device, operated by a content curator. The content curator maythen interact with the user interface to confirm that the keywords,metadata, and/or associated attributes, such as the definitions, arecorrect in the context of the text. A content curator may further reviewthat the keywords, metadata, and/or associated attributes areappropriate in the context of the text.

Text Difficulty Calculator

Systems and methods of language learning may include a text difficultycalculator module, which is a component of a computer program productembodied on a computer-readable medium and executed by a processor toperform the specified functionality. A text difficulty calculator modulemay determine a language difficulty for text in a resource.

In some embodiments, a text difficulty calculator may group resourcesinto a plurality of groups or proficiency levels according to difficultyscores. In such embodiments, resource groupings may correspond toproficiency levels assigned to language learners. That is, languagelearners may be ascribed a proficiency level determining the difficultyof resources in which they are presented for language-learning purposes,e.g., a beginner or first-level language learner may interact with firstlevel resource.

For example, in such embodiments a text difficulty score may identify aresource as having text of minimal difficulty relative to the rest of acorpus comprising one or more other resources stored in a resourcestore. This relatively easy resource may be grouped with resourceshaving comparable text difficulty scores. The group may now beconsidered a beginner level, a first level, or other starting leveldesignation.

In some embodiments, the number of groups, and/or the granularity towhich resources are segmented into these groups, is the decision of thecontent curator. In other embodiments, the determination of granularityand/or the number of groups may be automatically determined.

A text difficulty calculator module may analyze the text difficulty ofthe language in the text using a set of linguistic features. This set offeatures comprises one or more semantic features and one or moresyntactic features. Non-limiting examples of these features may includethe difficulty of the words in the text, the manner in which the wordsare used within the context of the overall text, the overallcomplication of the content conveyed by the text, and the difficulty ofthe syntactic construction of the text. Non-limiting examples ofdifficulty of syntactic construction may include the number of relativeclauses in the text, or the distance of pronouns from their relatedantecedents.

In some embodiments, a text difficulty calculator may identify syntacticand semantic features of the text in a resource. Using the identifiedsyntactic and semantic features as inputs into one or more algorithms,the text difficulty calculator may determine the language difficulty ofthe text. Some embodiments may implement a clustering analysis topredict the text difficulty of a given resource for particular groups ofnon-native speakers.

In some embodiments, a text difficulty calculator module must be trainedto be capable of identifying features in text and associate weightedvalues to features. The text difficulty calculator module may receivetraining resources having text in which various features are found.Training resources may be tagged or labeled with metadata indicatingweighted values associated with each of the features in the text, andthe resource may be tagged or labeled with metadata indicating the textdifficulty score of a training resource.

As an example, the text difficulty module may receive a set of trainingresources. Each of the training resources is labeled with dataindicating the difficulty score, e.g., on a scale from 1-7. For eachtraining resource in the set of training resources, the text difficultymodule may then extract a set of linguistic features having a weightedvalue. So for example, linguistic features in the text of a trainingresource may be associated with a floating point decimal representation.Taken together, the set of linguistic features identified in a trainingresource may be represented as a vector of numeric values based on theweighted values of each of the linguistic features. The vector for eachtraining resource in the labeled data set of training resources is thenused to train a statistical model for predicting a text difficultyscores of new resources. The statistical model may represent thecorrelation between the features selected for extraction from resources(e.g., identified in a list of lexical features to extract), theweighted values associated with those linguistic features, and thecorresponding labels. When new resources are received, a vector may bedetermined for the new resource based on linguistic features identifiedand extracted from the new resource. The vector of the new resource mayallow the system to label the new resource with a predicted textdifficulty score using the statistical model.

FIG. 3 shows a flowchart for an exemplary embodiment of a method ofdetermining a text difficulty score for language of text in a resource.

The embodiment of FIG. 3 shows steps S301, S303, S305, S307, S309, andS311. However, a person having ordinary skill in the art wouldappreciate that other embodiments may vary the steps performed. Theembodiment of FIG. 3 scans text to identify syntactic and semanticfeatures of the text, and then uses the identified features as inputsinto one or more algorithms to determine the language difficulty of thetext.

In step S301, the text difficulty calculator receives a resourcecontaining text and then the text of the resource is scanned for variousfeatures, as discussed in a next step S303.

Continuing with step S301, in some embodiments, a resource may bereceived or fetched from a resource store within a language learningsystem. In some embodiments, resources of various types may be receivedfrom a user interface on client computing device. This user interfacemay be that of a content curator or a language learner, and the userinterface may transmit one or more selected resources to thelanguage-learning system to implement in the various componentsdescribed herein. Resource received from a user interface may bereceived by a text difficulty calculator to perform a text difficultydetermination.

In some cases, a text difficulty calculator may scan only a portion, orseveral non-contiguous portions, of text of a resource. For example, thetext difficulty calculator may scan only a particular entry of anencyclopedia, without scanning the entire encyclopedia.

In a next step S303, the text difficulty calculator compares scannedtext against features listed in a feature bank.

A lexical feature bank may be a computer file containing a listing ofsemantic features and a listing of syntactic features. The featureslisted in the feature bank may be used to determine language difficultyof text in a resource. The features in the feature bank describe andcorrespond to various aspects of texts that make reading orunderstanding a text difficult for non-native speakers andsecond-language learners.

In a step next S305, a text difficulty calculator may identify a set offeatures in the text of a resource based on the comparison with thelexical feature bank in step S303. This set of identified features maycomprise semantic features and syntactic features.

Semantic and syntactic features identified in text may be features thatmake the text difficult to comprehend for language learners. Forexample, an adult language learner, who is capable of proficient readingcomprehension in their native language and also understands concepts mapto text, may be learning English. In this example, the learner onlyneeds to learn to read in English, but they do not need to learn thatconcepts map to the text written in English.

So in this example, semantic and syntactic features identified duringthe comparison step S303, are features of a text, written in English,that focus on the linguistic aspects of English that makes readingEnglish difficult. Or, the features may relate to concepts in Englishthat make English different from the language learner's native language.Non-limiting examples of this may include negative-polarity items (i.e,terms such as “anyone” or “no one”), which can be unique to English andmay be difficult to explain to a non-native language learner. Anotherexample may be a large amount of wh-movement, i.e., relative clauses faraway from the terms being modified by the relative clause.

In a step S307, after identifying a set of features in a text, eachsemantic feature and each syntactic feature in the set of features, areassigned a value or weight, based on a relative difficulty that eachfeature contributes to the overall difficulty of the text.

In some embodiments of a text difficulty calculator, a lexical featurebank may result from features extracted from a labeled data set oftraining resources. The lexical feature bank may be a file listinglexical features and is stored in a non-transitory machine-readablestorage medium accessible to the text difficulty calculator.

For example, a text difficulty calculator may employ a labeled data setof training resources with pre-identified semantic and syntacticfeatures label with metadata identifying the features in the text.Non-limiting examples of the features may include: lexical frequency ofvarious words (i.e., how often words show up in resources in theresource store corpus or in external corpora); the length of thesentences in the text; the amount of coordination and subordination inthe sentences; and the distance between pronouns and antecedents.

As previously mentioned, a labeled data set may be training resourceshaving metadata associated with the semantic and syntactic features inthe text of the training resource. In the labeled data set, each of thesemantic and syntactic features may be associated a weight correspondingto how much difficulty the particular feature contributes to the overalllinguistic difficulty of the text. When an extracted feature of text ina resource is identified as matching a particular feature in the labeleddata set, the text difficulty calculator may assign a particular weightassociated with the feature as found in the labeled data set.

In some embodiments, a content curator may determine each of theassigned weights. In some embodiments a text difficulty calculator mayaccept a manual input for each of the weights from a user interface of acontent curator. In some embodiments, a content curator may review,revise, and/or update the weights for each of the features. And in someembodiments, the content curator may use the user interface to amend oneor more algorithms, such as a logistic regression algorithm, todetermine the correct weight for each of the features.

For example, a user interface of a content curator may be used tocorrect a difficulty score calculated for a particular word, therebyupdating a weight automatically associated with a feature or word. Asanother example, a user-labeled weight (representing difficulty of afeature or word) may be manually entered as a training label to improveaccuracy of algorithms (e.g., a statistical model) used to automaticallycompute the text difficulty score

In a step S309, a text difficulty score is determined using the assignedweights for each of the features as parameters to one or more algorithmsused to determine the text difficulty score. Some embodiments may assigntwo values as weights, that is a first value is assigned and then asecond value may again be assigned, before determining the textdifficulty score.

For example, some embodiments may use an algorithm, such as aprobabilistic statistical classification model (e.g., a logisticregression algorithm), to determine correct weights for each of thefeatures in the set of features identified in resources within atraining corpus comprising one or more training resources. A firstweighted value may be assigned during lexical feature extraction ofsteps S301-S309. Each of the first values assigned to the features thenreceive a second value determined through analysis of the features ofthe text. Therefore, in this exemplary embodiment, the final textdifficulty score is determined using each of the first weighted valuesobtained during feature extraction and each of the second weightedvalues determined through analysis of the text.

In some embodiments, as in the present example, the textual features ofa resource logically separates text of resources into various weightedscores based on a difficulty determined for each identified feature.Then, an algorithm, such as a logistic regression model, converges theweights across a plurality of resources in a corpus such that weightsfor the features are determined based, at least in part, on theirstatistical distribution across resources in the corpus.

In a next step S311, the resource is grouped with resources of similardifficulty based on the text difficulty score calculated for the text.This step may categorize resources based on text difficulty score into aleveling system corresponding to a leveling system classifying languagelearners' proficiency in a relevant language.

In some embodiments, resources are grouped by proficiency level based oncomparable text difficulty scores. In such embodiments, after the textdifficulty score is calculated, resources are grouped on a scalecomprising various thresholds in a leveling system used for categorizingtext difficulty of resources. For example, a first level comprising aset of corpora having text difficulty scores in the range of ‘A’ to ‘C’;or a Level ‘A’ comprising a set of corpora having text difficulty scoresin the range of ‘0.1’ to ‘0.3’ on a global scale from 0 to 1.

Some embodiments of the text difficulty calculator may implement aclustering analysis to group the various resources by difficulty levels.After assigning weights to each of the textual features of resources ina corpus, the assigned weights for the textual features of a givenresource effectively separate the text of the resource into variousdifficulty categories according to each identified feature, then alogistic regression algorithm converges the weights to categorize theoverall resource into an appropriate difficulty category. Someembodiments may implement a principal component analysis to determine anoptimal number of clusters based on the variance.

In an example of the text difficulty calculator using a clusteringmethod, the clustering method used is a k-means methodology in which theinputs are the number of clusters implemented and a combination ofsyntactic and semantic features computed from the resource. Thismethodology may also determine the optimal number of proficiency levelgroups, or clusters, of resources in which to categorize resources in aresource store.

In this example, the features of the text may include a proportion ofwords in the resource that may be found in the Academic Word List,adjective variation (AdjV), adverb variation (AdvV), bilogarithmic typeto token ration (B_TTR), lexical word variation (LV), modifier variation(ModV), noun variation (NV), average number of characters (NumChar),average number of syllables (NumSyll), squared VV1 (SVV1), Uber Index(Uber), and verb variation-I (VV1).

In another example, embodiments of the text difficulty calculator modulemay comprise two components: (a) a program derived from sckit-learn forclustering analysis, and (b) syntactic complexity code. The textdifficulty calculator may use a known scripting language, such as Pythoncoding, to extract the set of semantic and syntactic features and anatural language tool kit for part-of-speech tagging to effectuateword-feature identification and text difficulty calculation.

Distractor Generator

Some embodiments of the system or method of language learning mayinclude a distractor generator module. A distractor generator module isa component of a computer program product embodied on a machine-readablemedium and executed by a processor to perform the specifiedfunctionality.

In some embodiments of a language learning system, distractors may beused in customized language-learning lessons for a language learner. Thedistractor generator module may automatically identify and generatesyntactic, orthographic, phonetic, and semantic (synonym and antonym)distractors.

Those embodiments of a language learning system implementing adistractor generator may include a computing device of a languagelearner, which may connect to a distractor store to request distractorswhen a certain word is being tested. A distractor store may be adatabase comprising a non-transitory machine-readable storage mediumstoring one or more distractors generated by the distractor generator.

Embodiments of a language learning system implementing a distractorgenerator may include keyword store storing keywords extracted from textof resources by the language learning system. Or, in some cases,keywords may be input from a user interface of a computer device of alanguage learner, content curator, or other user such as anadministrator.

Embodiments of a distractor generator may access one or more dictionarysources to identify a rough approximation of a definition of a word toreturn appropriate and effective distractors. A dictionary source may bea keyword store of the language learning system built by keywordsextracted from resources stored in the resource store. A dictionarysource may be another database in the system storing words in associatedwith a definition. A dictionary source may be an externally referencedsource, such as a website or a commercial service providing searchableaccess to words and their associated definitions.

Embodiments of a distractor generator may use a number of differentsources and tools as dictionary source. For example, a dictionary sourcemay be a licensed dictionary software tool having definitions and, insome cases, pronunciation data. Commercially-available dictionary toolsmay also be dictionary sources used, such as WordNet, which is adictionary software tool representing relationship between clusters ofwords. And, in some cases, the keyword store may be implemented as adictionary source; but, embodiments of the keyword store may havemetadata attached to each entry, like audio pronunciation, which may begenerated through associated recorded speech. Moreover, metadata mayrelate distractors and keywords in the keyword store back to resourceswith which they are associated.

Some embodiments of a distractor generator may use a heuristic wordsense disambiguation function, which may be aided by a natural languageprocessing toolkit of a known computer scripting language, such asPython or Java. The distractor generator may then identify a similardefinition in a dictionary source such as a keyword store or an externaldictionary source such as an online computer-searchable OxfordDictionary.

Exemplary embodiments of a distractor generator module may include suchcomponents as a natural language processing toolkit for part-of-speechtagging, pyEnchant, aspell, Oxford ESL Dictionary, and/or WordNet.

FIG. 4 shows a flowchart of an exemplary embodiment of the methodexecuted by the distractor generator module.

Embodiments of a distractor generator module may automatically identifyand generate different types of distractors for a target word. In someembodiments of a language learning system, the distractors generated bya distractor generator may be implemented in for various pedagogicalpurposes in language learning, such as evaluations and/or educationalactivities.

In a step S401, a distractor generator receives a target word from amodule or component within a system, which may be a language learningsystem. A distractor generator may receive a single target word or aplurality of target words, for which distractors will be automaticallygenerated.

In some embodiments of a language learning system, a distractorgenerator may receive a set of keywords extracted from a resource. Theset of keywords comprising one or more target words for whichdistractors may be automatically generated. The distractor generator mayautomatically generate a set of distractors for each of the targetwords.

In a step S402, the distractor generator module generates a set ofsemantic distractors related to the target word. The set of semanticdistractors may comprise one or more synonyms of the target word.Additionally or alternatively, the set of semantic distractors maycomprise one or more antonyms of the target word.

Embodiments of a distractor generator may use natural languageprocessing, implemented by, for example, a natural language processingtoolkit of a known computer scripting language, to associate similarwords and/or definitions from a dictionary with the target word, to aidlanguage learners to better master the target word.

A distractor generator may search dictionary sources to identify one ormore synonyms of the target word and one or more antonyms of the targetword. A dictionary source may be a keyword store, another dictionaryservice within the language learning system (e.g., a manually updatedtext file), and/or a dictionary computer service external to the system(e.g., the Merriam-Webster's® website). The distractor generator maygenerate a set of semantic distractors by choosing one or more of theidentified synonyms and/or one or more of the antonyms. The set ofsemantic distractors may be words to aid language learners as theymaster what the target word means. Some embodiments of the languagelearning system may implement the semantic distractors to distractlanguage learners from the correct meaning of the target word. Thedistractor generator may automatically generate the set of semanticdistractors to aid language learners understand a definition of thetarget word.

In a next step S405, a distractor generator may identify a set oforthographic distractors for the target word. Orthographic distractorsmay be words having a small edit distance with respect to a target word.An edit distance is the total number of letter-insertions and/orletter-deletions required to have one word resemble a second word. Forexample, changing the word “could” to “cold” is an edit distance of 1because only a letter deletion was required to change “could” to “cold.”Likewise, in another example, changing “could” to “would” is an editdistance of 2 because one letter is deleted and one letter is inserted.And, in another example, changing “could” to “should” is an editdistance of 3 because one letter is deleted and two letters areinserted.

Using words found in one or more dictionary sources, the distractorgenerator in step S405 may locate words having a minimal edit distancein relation to a target word that is going to be exercised. That is, thedistractor generator may identify an edit distance from the target wordto each of the words in a dictionary source. The distractor generatormay rank the words of a dictionary according to smallest edit distance.A distractor generator may then start from a word that is ranked withsmallest edit distance and then walk through the ranked words towardsword having larger edit distances until the set of orthographicdistractors is generated. The set of orthographic distractors compriseswords having small, or the smallest, edit distances satisfying adifficulty level related to the abilities of the learner.

The smaller the edit distance is between two words, the harder it may befor a non-native speaker to distinguish the two words from each other.In contrast, words having a larger edit distance may be easier todistinguish. Applying this to generating orthographic distractors,smaller edit distances between orthographic distractors and a targetword may result in comparatively more difficult distractors for languagelearners. The difficulty level of orthographic distractors may thereforebe adaptably varied by selecting words from the dictionary source havinglarger/smaller edit distances.

In some embodiments, selecting words having the smallest desired editdistance may be based on a pre-determined amount of words to be used.For example, if there are to be 10 comparatively difficult orthographicdistractors in the set, then 10 words ranked as being closest to thetarget word are selected.

As described above, in some embodiments, a minimum edit distance, amaximum edit distance, or an exact edit distance, may be used toselected words in the set of orthographic distractors based on a desireddifficulty level. The set of orthographic distractors may be determinedby a difficulty setting, which may allow the distractor generator toautomatically adapt complexity of orthographic distractors selected andthe amount of orthographic distractors in the set. For example, an exactedit distance of 3 may be used to generate a comparatively lessdifficult set of orthographic distractors. In this case, all of thewords identified as having an edit distance of 3 in a dictionary may beincluded in the set of orthographic distractors. The number oforthographic distractors may be limited by various means previouslydescribed, or may include all of the words satisfying the edit distancecriteria.

It is to be appreciated that other algorithms for selecting orthographicdistractors based on an edit distance in relation to a target word maybe used to generate a set of orthographic distractors. An ordinaryartisan would appreciate that other algorithms to automatically identifyand generate orthographic distractors based on edit distance may fallwithin the scope of one or more portions of the invention describedherein.

In a next step S407, a distractor generator may generate a set ofphonetic distractors related to a target word. A phonetic distractor maybe a word tending to cause distraction based on the phonetic similaritybetween words. Embodiments of the distractor generator may automaticallyidentify one or more phonetic distractors in one or more dictionarysources and then generate the set of phonetic distractors from one ormore of those identified.

Phonetic distractors may be words that sound similar to a target wordbut are not intended to be identically pronounced when spoken. Aphonetic distractor may be a word that sounds similar to the targetword, but may also have a small edit distance in relation to the targetword. Some embodiments of the language learning system may utilize theset of phonetic distractors in pedagogical learning activities where,for example, language learners must choose between two different itemsthat sound similar to one another.

Some embodiments of a distractor generator may comprise a step S408 inwhich words that sound identical, or homophones, are excluded from theset of phonetic distractors. In such embodiments, a distractor generatormay recognize a word as being a homophone of a target word. For example,in some cases, a homophone may be included to a set of orthographicdistractors because the homophone has an edit distance meeting thecriteria for orthographic distractors; meaning the homophone could beincluded in the set of phonetic distractors. But embodimentsimplementing a step S408, or the like, may identify homophones andexclude them from the set of phonetic distractors.

In a next step S409, a distractor generator may identify a set ofsyntactic distractors. A syntactic distractor may be a word related to atarget word but differs in some grammatical fashion, e.g., the word is adifferent conjugation of a verb, the word is a verb that agrees with adifferent person being referenced (first-person, second-person,third-person), the word is a noun form of a verb, or the word is a verbform of a noun.

In a next step S411, a distractor generator may store each of the setsof distractors into a distractor store. In some circumstances, step S411may automatically update an existing set of distractors or receive aninput from a user interface of a content curator's computing devicechanging one or more of the distractors.

In a next step S413, a distractor generator may output one or more ofthe automatically generated sets of distractors to a user interface of acomputing device.

In some embodiments, the user interface receiving the output of thedistractor generator may be that of a content curator who may reviewautomatically generated distractors for accuracy and consistency. Shouldthe content curator wish to amend a set of distractors, such embodimentsmay provide for the content curator to make necessary changes via theuser interface.

In some embodiments, the user interface receiving the outputteddistractors may be that of a language learner who may reviewautomatically generated distractors for accuracy and consistency;particularly where the learner was the originator of the resource fromwhich words was extracted. Should the learner wish to amend a set ofdistractors, such embodiments may provide for the learner to makenecessary changes via the user interface.

In some embodiments, distractors or sets of distractors may beassociated with keywords through relational metadata. That is,distractors may be stored in a distractor store with metadata thatassociates keywords to particular distractors. Similarly, in someembodiments of a keyword store, as described herein, keywords may bestored with metadata the associates distractors with particularkeywords.

Learning Activities and Activity Sequences

Embodiments of a language learning system may include a learning modulelessons in which the objective is having a language learner demonstrateknowledge of a concept or helping a learner learn a concept byperforming specific learning activities.

A learning module may generate a learning activities of various typesthat are designed to foster various language skills such as reading,writing, listening, and speaking proficiency. The various activity typesmay also focus on vocabulary building, grammatical proficiency, and/orpronunciation, among other skills. Examples of activities may include amultiple choice question activity, a vocabulary matching activity, aspeaking focused activity, a pronunciation focused activity, a writingfocused activity, a grammar focused activity, a listening focusedactivity, a reading focused activity, a spelling focused activity,identifying a vocabulary word activity, an understanding of audioinformation activity, an understanding of video information activity,and a reading comprehension activity.

A learning module may dynamically generate a lesson that may comprise anumber of learning activities tailored for language learners. Whendynamically generating a learning activity, the language module mayutilize output generated by a keyword extractor, a distractor generator,and/or a text difficulty calculator.

In some embodiments, learning activities may be customized based on userdata associated with learners that is stored in a user data store. Forexample, learning activities may be varied by difficulty to correspondto a language learner's proficiency level. Other variations based onuser data may include customizations based on a language learner'spersonal goals, needs, and performance. The learning module mayautomatically utilize output from various modules and data stored indatabases to, in turn, automatically generate learning activities suitedfor a language learner.

The system may set forth a pedagogic path for each level of languagelearning, whereby each pedagogic path comprises a series of learningactivities. The learning activities can vary based upon the learner'sneeds and proficiency level, as well as different constraints, such as anumber of words used in an activity, a time constraint, whether to useaudio or textual hints (e.g., revealing part of a word). The system candynamically build an appropriately difficult activity based upon thetype of learning activity required by the learner's past performance.

An exemplary activity for a learning activity may be a vocabularyactivity in which the learner identifies synonyms of a given word. Thesynonyms used in the learning activity may be chosen based upon theneeds and proficiency of the learner. The learning activity types areestablished within the system, but the content and words are dynamicallyadjusted for each learner and allow for a customized activity.

Each learning activity may use distractors. The system choosesappropriate distractors based on their type and relationship to thetarget word for use in different activities, including multiple choicequestions, matching activities, spelling activities, activities forreconstructing a text, and memory games. The learning activity modulemay score a learner's performance and automatically assign a grade tothe learner for one or more skills in an activity.

At a higher performance grouping level, more difficult distractors maybe used. Learning activities may be automatically adapted within alesson, or for a next lesson, in response to a change in the learner'sskills and/or overall proficiency. In some embodiments, the learningactivity module may update the learner's profile in the user profilestore to reflect grades and changes in the learner's abilities.

For example, a lower-level learner might have to complete a blank spacewith a correct word and be presented with options such as the targetword, an antonym, and another word with a similar spelling. Ahigher-level learner performing the same activity might haveincreasingly difficult options such as more distractors and/or moresimilarly spelled words.

A lesson may comprise a series of learning activities that may bederived from the combinations and permutations corpora, keywords, anddistractors. Each lesson relies on resources, such as a document, andincludes a specific series of learning activities that may increase indifficulty. For example, an activity sequence focused on reading andspelling would begin with an initial reading comprehension activity,then move to a vocabulary acquisition activity, and finish with one ortwo spelling activities. As the learner demonstrates proficiency andimproved performance, the keywords targeted can get increasinglydifficult.

The difficulty level of a given learning activity may be determined by adifficulty level (which is chosen based on a user's proficiency, i.e.,performance on assessment) and learning activity difficulty level (whichis chosen based on a user's performance within lessons/courses). Thelearning activity difficulty level may be related to the text difficultylevel of the underlying resource used in the learning activity. The textdifficultly level may be determined with an algorithms using metricssuch as a readability index, average sentence length, number of wordsfrom Oxford 3 k, number of words from the AWL, number of subordinateclauses, number of relative clauses, number of common simple tenses,number of common progressive tenses, number of common future tenses, orwords tagged in an image.

For example, a relatively high text difficulty score for a resource mayindicate more difficult or complicated language in the text because textcontains words that appear less frequently, longer sentences, academicwords, and high relative frequency of complex forms.

A relatively high learner proficiency score indicates that a learner isvery proficient in one or more language skill domains, such as reading,writing, listening, and speaking. In other words, a learner with a highproficiency score is said to have high ability scores in variousspecific language skills. Learners having a relatively high learnerproficiency score may be able to handle resources, and content inresources, having more difficult language, indicating that such learnershave a robust vocabulary, a strong command of grammar, and strongperformance across language skill domains, i.e., high abilities inspecific language skills. Resource difficulty levels, such as a textdifficulty score, may be mapped to learner proficiency levels so thatall resources and learning activities that learners presented withgenerally contain language at a level of difficulty that is at, orsometimes slightly above, a learner's current level.

Embodiments of a learning activity module may implement variousmultimedia sources, including audio input or output, video input oroutput, textual input or output, and image input or output. A learner'sresponses to a learning activity may be saved in a user data store forassessing the user's performance and/or for feedback from a tutor orcontent curator. In some learning activities, a learner may receive aresource having errors in the text, and be required to edit itthemselves. The learner may then submit this completed learning activityfor asynchronous offline feedback.

Learning activities and each type of learning activity may be modifiedto be more or less difficult while holding the underlying resourceconstant. Learning activity difficulty may be influenced by, forexample, the difficulty of distractors that are used. For example, an“easy” version of a learning activity would use “easier” distractorsthan the “difficult” version. Other non-limiting examples of how thedifficulty of learning activities may be varied may include adjustingthe number of items tested, the presence/absence of a timer, and/orother activity-specific variations such as longer words in a type ofactivity that is vocabulary focused. In some embodiments, learningactivity difficulty may be manually modified or automatically adaptedfor learners within their coursework tracks and lessons.

Adjusting difficulty may also be based on a word difficulty. Words mayhave difficulty scores associated with them. In some cases, words may benormalized on a zero to one scale so that they may be associated withlearning activities appropriately. Inputs into word difficulty scoringmay include frequency within a variety of corpora, such as the AmericanNational Corpus, as well as metrics such as number of syllables andnumber of definitions for the word. Word difficulty levels may be usedto sequence learners' exposure to words within a lesson/learningactivity sequence. Learners' word mastery scores may be used todetermine which words to feature in various learning activities.

Examples of the Graphical User Interface for a Learner

FIG. 5 shows a screenshot of an exemplary embodiment of a graphical userinterface (GUI) 500 displaying a home menu before a lesson 505, to alanguage learner, on a monitor of the learner's computing device. TheGUI screenshot 500 comprising a diagnostic score 501, a languageproficiency level of the user 502, a learning focus 503 on a particularlanguage skill, a goal 504 for the coursework, a lesson 505, and a startbutton 506. The screenshots 600, 700, 800 are examples learningactivities of various activity types, which were dynamically generatedfor the learner; together they may comprise a lesson 505.

An activity in this exemplary lesson 505 may be reading comprehension,as shown in FIG. 6. Another activity in this exemplary lesson 505 may bea vocabulary activity, as shown in FIG. 7. Another activity in thisexemplary lesson 505 may be an activity involving spelling withdifferent subset of keywords, as shown in FIG. 8.

FIG. 6 is a screenshot of a GUI 600 for a learner to engage in a readingcomprehension activity prepared by the language learning system. Theexemplary GUI screenshot 600 for a reading comprehension activity maycomprise text 601 of a document resource that is used for the exemplarylesson 505, a document title 602, one or more highlighted keywords 603,and a reading comprehension quiz 604 comprising a question 604 a and aset of multiple choice answers 604 b. The exemplary screenshot of a GUI600 may highlight various keywords 603 and presents a comprehension quiz604 that implements the keywords 603 and a set of distractors in theform of multiple choice answers 604 b.

FIG. 7 is a screenshot of a GUI 700 for a user to engage a vocabularyactivity prepared by the language learning system. The exemplary GUIscreenshot 700 for a vocabulary activity may comprise text 701 of adocument resource, a document title 702, a redacted word 703 in the text701, and a word match question 704. The word match question having a setof multiple choice answers 704 b in which the correct answer correspondsto the redacted word 703.

FIG. 8 is a screenshot of a GUI 800 for a user to engage a spellingactivity prepared by the language learning system. The exemplary GUIscreenshot 800 may present an activity involving spelling with differentsubset of keywords. The GUI screenshot 800 comprising text 801 of adocument resource, a document title 802, a redacted word 803 in the text801, and a word scramble quiz 804 comprising a question prompt 804 a anda scrambled set of letters 804 b.

Example of a Learning Activity Module

FIG. 9 shows a method of a language system implementing a learningmodule according to an exemplary embodiment.

The exemplary method embodiment of FIG. 9 comprises steps S901, S905,S911, S904, S906, S908, S910, S920, S921, S922, S923, S924, S925, andS926, and may implement a user data store 902, a keyword store 907, aresource store 909, and a distractor store 913.

In a first step S901, a language learning system may receive a newdocument resources comprising text. The new text may be received from acomputing device of a language learner (“learner”). The new text may beinput from a computing device of a content curator. The new text may beautomatically downloaded or transmitted from a text-producing source,such as, for example, a website, a blog, a news outlet, or a textbookpublisher.

In a step S903, a keyword extractor module implemented by the system mayextract one or more keywords from the new text. The keywords may beextracted according to an algorithm adapted to identify and extractkeywords that are pedagogically valuable words effectuating languagelearning. The keyword extractor module may be adapted to determine aword difficulty score for a keyword. The keyword extractor module may beadapted to associate one or more other attributes to a keyword, such asword length, number of syllables, and/or a part-of-speech.

A keyword extractor module may store the extracted keywords in a keywordstore 907, which is non-transitory machine-readable storage mediumstoring keywords. A keyword store 907 may store data associated with thekeywords, such as word attributes, a word difficulty score, sourcedocument, and/or keys associated with the keywords stored in the keywordstore 907.

In some embodiments, the keyword extractor module may extract and storedifferent keywords depending upon a profile of the learner. Keywords maybe assigned a more difficult keyword score if they, for example, areassociated with an esoteric or unique definition. Keywords may also bemore difficult if they have are longer by number of letters andsyllables. Keywords may be more difficult if they have silent letters,abnormal pronunciations, or non-intuitive spellings. In suchembodiments, a keyword extractor may generate an additional word score(aside from the word scoring done to determine candidate keywords) torank words in terms of likelihood of difficulty for non-native speakersusing word length and word and part-of-speech frequency in externalcorpora. As discussed below, a learning activity may be made moredifficult by using distractors that are intended to test more difficultkeywords extracted from a text.

In a next step S905, a text difficulty calculator module may determine atext difficulty score for the text of a new resource. In the exemplaryembodiment, a new document resource comprising text is received, thetext difficulty score is determined, and then the new resource may bestored into a resource store 909. The resource store 909 may be anon-transitory machine-readable storage medium storing one or moreresources. In some embodiments, the resource store 909 may also storemetadata associated with resources that describe various attributes ofthe associated resources, such as a text difficulty score. In someembodiments, a document may already be stored in the resource store 909,in which case step S905 would not be necessary.

In a step S911, a distractor generator may generate a set of one or moredistractors. Distractors may be generated and stored into a distractorstore 913. Distractors may be of various different types, each of thetypes are designed to exercise language learners' abilities inparticular language skills, i.e., spelling, vocabulary, phoneticdistinctions.

In some embodiments, the distractor generator may generate as manydistractors as needed to test each of the extract keywords. Distractorsmay be of varying difficulty based on a number attributes of adistractor, such as, for example, the difficulty of the underlyingkeyword being testing, or the closeness of the keyword and distractor.

Some embodiments of a learning module may utilize a user data store 902,which a non-transitory machine-readable storage medium that stores userprofiles associated with language learners. The user profiles maycomprise information about language learners, such as a learner goal forthe learning the language, a subject matter interest for contentcontained within text, ability scores for language skills, and/or thelearner's overall proficiency level at the language.

In a step S904, a learning module may identify a learner proficiencylevel in the user data store 902. The proficiency level may be anoverall valuation of the learner's skill level, whereas an ability maybe for a particular skill. A proficiency level may be determined byamalgamation of assessments individual abilities, or may be determinedby known means of assessing a language proficiency level.

In a step S906, a learning module may identify the need for targetedlanguage skill practice based on the scored abilities of the learner inthe user data store 902. An example of a language skill may be readingcomprehension or spelling. An ability score may be an indicator of thelearner's ability to perform that particular language skill. Thelearning module may make use of any number of language skills whenpreparing learning activities. A learner profile may store an abilityscore for each of the language skills exercised by the learning module.

In a step S908, a learning module may identify one or more goals thatthe learner has for learning the language. A goal may be, for example,studying for the TOFEL examination, preparing for a tourism vacation,preparing for a business trip, school course, professional requirementor military deployment. A learner may be associated with more than onegoal. A goal may be a pre-determined list from which the learner mayselect, or a goal may be input from the learner at a prompt. A goal mayalso be input by a content curator or other administrator.

A learning module may prepare learning activities relative to a learnerbased on the learner's needs, i.e., weaker abilities in specific skills.And, in some embodiments, the learning module may prepare lessons sothat learners may achieve certain goals. That is, the learning modulemay track learners' progress of their proficiency level and abilities inspecific skills relative to their stated goals. A goal may be met when alearner's proficiency level and/or abilities reach a level comparable totheir goal. Learners having a goal demanding a higher proficiency level,such as preparing for the TOEFL, may receive a longer set of lessonand/or a more comprehensive set of lessons, as compared to learners whohave a less demanding goal, such as learning basic vocabulary for casualconversation.

For example, two learners, who are at the exact same proficiency level,may have different goals in which they wish to attain. The firstlearner, for example, may only want to be able to order coffee in thenew language, in which case the first learner would be able to reachthat goal more quickly and the content the first learner would interactwith would generally relate to ordering food and drink. If a secondlearner, at the same proficiency level, wanted to become fluent in thenew language, then the second learner would have a longer collection ofcoursework (e.g., longer lessons, and/or more lessons). In this example,both the first learner and the second learner might begin with learningactivities directed towards ordering coffee or ordering food and drink,because those were learning objectives that both learners wanted to beable to accomplish. The second learner, however, would then interactwith learning activities involving increasingly difficult languageskills. In cases, the second learner would receive longer lessonscomprising more learning activities as compared to the first learner.

Some embodiments of the learning module may allow learners to take anachievement test to determine whether learners have met their statedgoal. Learners' proficiency levels and abilities may be scored andtracked throughout their interaction by the language learning system. Insome cases, the learning module may inform users when they reach aproficiency level comparable to their stated goals. Learners may take anachievement test that assesses whether learners have achieved theirgoals.

In a next step S910, a learning module may identify a learner's contentinterest of the subject matter of potential resources, according to thelearner's interests stored in the user data store 902. A learner'sinterest may be, for example, a sport. These may be input from a userinterface of the learner in which the learner lists interests.

In next steps S920-S926, a learning module selects a resource to utilizefor generating a series of learning activities, where the series oflearning activities may be a lesson. The lesson may be a set of learningactivities generated using the identified learner attributes from stepsS904, S906, S908, and S910, and also using the learning activitybuilding blocks generated in steps S901, S903, S905, and S911.

In a next step S920, a resource may be selected from the resource storebased on one or more learner attributes. A resource may be selectedaccording to any permutation of learner attributes, such as a learner'sproficiency level, goals, and interests.

In some embodiments of the learning module, learning activities may begenerated to automatically include a resource containing content that isrelevant to the learner's interests. For a document resource, thecontent of the text may be identified using metadata associated withextracted keywords. The content of the text may also be identified bysome other known natural language processing technique that may identifyand/or categorize subject matter of a resource, a corpus, or othercollection of resources.

As an example of delivering certain resources based on their content, alearner with an interest in soccer may interact with learning activitiesrelated to resources describing a recent soccer match, text explainingthe rules of soccer, or text detailing the history of a famous soccerclub.

As previously mentioned, some embodiments of a learning module mayselect resources based on how complicated their content is. That is, insome cases, resources' difficulty scores may be based, in part, on howcomplicated their content is. For example, if a novice learner is aphysicist with an interest in complicated scientific topics, then textwhose content discusses the complicated scientific topics in thelanguage being learned, may be too difficult to effectively teach thenew language to the novice learner. Thus, some embodiments of a learningactivity module may use a text difficulty score to determine anappropriate resource to fetch from a the resource store when building alearning activity.

Embodiments of the learning activity module may vary permutations oflearners' attributes and permutations of learning activities' buildingblocks are used when generating learning activities.

In a next step S921, a learning activity module may target a particularlanguage skill to exercise in the learning activity being generated. Theparticular skill being targeted is based on the learner's abilities inindividual language skills. A learning activity module may adaptablyconstruct a series of learning activities that employ activitiesaddressing weaknesses in the particular skills of the learner.

Embodiments of a learning activity module may also adaptably constructlearning activities based upon the goals of the learner. For example, ifthe learner is a tourist, then learning activities may compriseactivities focusing on simpler vocabulary about landmarks or directions.In a contrasting example, a learner studying for the TOEFL may receivemore difficult grammatical activities.

Next, in a step S922, by targeting a particular language skill in stepS921, the learning activity module may determine which type of activitythe learning activity must employ to appropriately address theparticular language skill. For example, when a learning activity moduledetermines that a learner must target phonetic understanding skills, thelearning activity module may automatically generate a phonetics-basedactivity.

Non-limiting examples of activities may include: a reading comprehensionactivity (e.g., a multiple choice question about the content of a corpusor resource just consumed), a word match activity (e.g., fill in theblanks in an article choosing from list of all available keywords), avocabulary challenge (e.g., show a definition and answer multiple choicequestions to identify the target keyword, in text or with audio), asound drop activity (e.g., in a range of text, have missing words andask user to listen to multiple audio files to find the match and drop itin) a memory game activity (e.g., have cards with keywords andsynonyms/definitions covering the original content and flip them overconcentration style to match the words), and a word scramble activity(e.g., see a word/phrase blanked out in a range of text and have theletters re-arranged so it has to be spelled out by clicking on them inorder).

In a next step S923, a learning activity module may select a set ofdistractors, derived from the selected resource, which are suited to atype of learning activity.

A learning activity module may identify and select distractorsappropriate for the type of learning activity determined for thelearning activity within step S922. For example, phonetic distractorsmay be selected when the activity type will be a sound drop activity.

A learning activity module may identify and select distractors having adifficulty level that is comparable to a learner's ability in aparticular language skill that is going to be exercised in the learningactivity.

In a next step S924, a learning activity module may generate a learningactivity using building block components of a learning activity andbased on learner attributes.

Building block components of a learning activity may be a resource, aset of keywords associated with the resource, an activity exercisingparticular language skill, and a set of distractors derived from adictionary source using a text's keywords (e.g., synonym semanticdistractors or phonetic distractors). Learner attributes may beinformation describing a learner's preferences and/or informationdescribing the learner's capabilities (e.g., an ability for vocabulary,or an overall proficiency level).

In a next step S925, a learning activity module may generate a lessoncomprising a set of learning activities. The lesson, as well as theindividual learning activities may be customized for a learner. A lessonmay provide learners a path toward accomplishing learning goals. Alearner profile may store milestones related to that path. A lesson maysequence learning activities to maximize a pedagogical value.

In a next step S926, a learning activity module may generate a unit ofcoursework comprising of one or more lessons. A unit may be a path foraccomplishing learning goals. Lessons may be sequenced to maximizepedagogical value. Content of resources and learning activities may bepersonalized and customized throughout a unit.

In some embodiments, a learner may have one primary active course,composed of goal-oriented tracks, and made up of language skill-focusedlessons using interest-focused resources. Units may allow users toattain a badge by passing optional achievement tests.

A learning activity module may also include synchronous or livetutoring. A learning activity module may also include asynchronous tutorfeedback. Asynchronous feedback may be incorporated as learningactivities in lessons that are tailored to a learner's needs. Livetutoring may be instituted in sessions that may be scheduled inconjunction with a learner's lesson or in sessions independent of alesson, which may occur at set intervals.

Unless specifically stated otherwise as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “creating,” “providing,”“calculating,” “processing,” “computing,” “transmitting,” “receiving,”“determining,” “displaying,” “identifying,” “presenting,”“establishing,” or the like, can refer to the action and processes of adata processing system, or similar electronic device, that manipulatesand transforms data represented as physical (electronic) quantitieswithin the system's registers or memories into other data similarlyrepresented as physical quantities within the system's memories orregisters or other such information storage, transmission or displaydevices. The system can be installed on a mobile device.

The exemplary embodiments can relate to an apparatus for performing oneor more of the functions described herein. This apparatus may bespecially constructed for the required purposes, or it may comprise ageneral purpose computer selectively activated or reconfigured by acomputer program stored in the computer. Such a computer program may bestored in a machine (e.g. computer) readable storage medium, such as,but not limited to, any type of disk including floppy disks, opticaldisks, CD-ROMs and magnetic-optical disks, read only memories (ROMs),random access memories (RAMs), erasable programmable ROMs (EPROMs),electrically erasable programmable ROMs (EEPROMs), magnetic or opticalcards, or any type of media suitable for storing electronicinstructions, and each coupled to a bus.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

Embodiments implemented in computer software may be implemented insoftware, firmware, middleware, microcode, hardware descriptionlanguages, or any combination thereof. A code segment ormachine-executable instructions may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

The actual software code or specialized control hardware used toimplement these systems and methods is not limiting of the invention.Thus, the operation and behavior of the systems and methods weredescribed without reference to the specific software code beingunderstood that software and control hardware can be designed toimplement the systems and methods based on the description herein.

When implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer-readable orprocessor-readable storage medium. The steps of a method or algorithmdisclosed herein may be embodied in a processor-executable softwaremodule which may reside on a computer-readable or processor-readablestorage medium. A non-transitory computer-readable or processor-readablemedia includes both computer storage media and tangible storage mediathat facilitate transfer of a computer program from one place toanother. A non-transitory processor-readable storage media may be anyavailable media that may be accessed by a computer. By way of example,and not limitation, such non-transitory processor-readable media maycomprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othertangible storage medium that may be used to store desired program codein the form of instructions or data structures and that may be accessedby a computer or processor. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media. Additionally, the operations of a method oralgorithm may reside as one or any combination or set of codes and/orinstructions on a non-transitory processor-readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the invention. Thus, the present invention is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the following claims and theprinciples and novel features disclosed herein.

While various aspects and embodiments have been disclosed, other aspectsand embodiments are contemplated. The various aspects and embodimentsdisclosed are for purposes of illustration and are not intended to belimiting, with the true scope and spirit being indicated by thefollowing claims.

What is claimed is:
 1. A computer-implemented method for predicting atext difficulty score for a new resource, the method comprising:extracting, by a computer, one or more linguistic features having aweighted value from a plurality of training resources containing text,wherein the text is associated with a metadata label containing a textdifficulty score of the text; determining, by the computer, a vectorvalue associated with each training resource based on each of theweighted values of each of the extracted one or more linguisticfeatures; training, by the computer, a statistical model using thevector values associated with each training resource, wherein thestatistical model represents a correlation between a set of featuresselected for extraction, a set of weighted values assigned to the set offeatures selected for extraction, and a set of text difficulty scoresassociated with the training resources; extracting, by the computer, oneor more linguistic features having a weighted value from a new resource;determining, by the computer, a vector value for the new resource basedupon the set of extracted linguistic features; and predicting, by thecomputer, a text difficulty score for the new resource based upon thevector value for the new resource and the statistical model.
 2. Acomputer-implemented method for determining text difficulty for aresource comprising: comparing, by a computer, at least a portion oftext related to a resource against a lexical feature bank filecomprising a listing of semantic features and a listing of syntacticfeatures; identifying, by the computer, a set of lexical featuresassociated with the text based on the comparison, wherein the set oflexical features comprises at least one semantic feature from the textmatching a semantic feature in the list and at least one syntacticfeature from the text matching a syntactic feature in the list;assigning, by the computer, a first value to each of the lexicalfeatures in the set of lexical features; and determining, by thecomputer, a text difficulty score for the text using each of the firstvalues associated with each of the features in the set of lexicalfeatures.
 3. The method according to claim 2, wherein the resource isone or more portions of a document resource containing the text.
 4. Themethod according to claim 2, wherein the resource contains and audiotrack, and wherein the text is a transcript of the audio track.
 5. Themethod according to claim 2, further comprising: receiving, by thecomputer, the resource from a user interface of a computing device; andparsing, by the computer, the text into one or more portions of text,wherein each portion of text is compared against the lexical featurebank.
 6. The method according to claim 5, further comprising receiving,by the computer, the resource from a resource store storing one or moreresources.
 7. The method according to claim 6, further comprising:receiving, by the computer, one or more training resources containingtext, wherein the one or more semantic and syntactic features of thetext are labeled with metadata indicating the features and an associatedfirst weighted value associated with each of the features; andgenerating, by the computer, the lexical feature bank comprising each ofthe semantic and syntactic features labeled in the one or more trainingresources and each of the associated first weighted values.
 8. Themethod according to claim 7, further comprising assigning, by thecomputer, a second value to each of the features in the set of lexicalfeatures, wherein the computer determines the text difficulty score forthe text using each of the first values and each of the second valuesassociated with each of the lexical features.
 9. The method according toclaim 2, further comprising outputting, by the computer, the textdifficulty score to a user interface of a computing device of a contentcurator.
 10. The method according to claim 9, transmitting, by thecomputer, a set of values comprising the first values of each feature inthe set of lexical features to the computing device of the contentcurator for the content curator to review.
 11. The method according toclaim 10, further comprising updating, by the computer, the textdifficulty score of the text according to a score adjustment receivedfrom the user interface of the content curator.
 12. The method accordingto claim 11, further comprising updating, by the computer, the set ofvalues based on one or more amendments received from the user interfaceof the content curator.
 13. The method according to claim 2, furthercomprising: comparing, by the computer, the text difficulty scoreassociated with the resource against one or more stored text difficultyscores associated with one or more resources stored in the resourcestore; and determining, by the computer, a skill level associated withthe resource according to the text difficulty score associated with theresource relative to the one or more stored text difficulty scores. 14.A system comprising a processor and non-transitory machine-readablestorage containing a text difficulty calculator module instructing theprocessor to execute the steps of: comparing text in a resource againsta listing of features comprising a list of semantic features and a listof syntactic features; identifying one or more semantic features in thetext matching the listing of features, and identifying one or moresyntactic features in the text matching the listing of features;assigning a value to each of the identified semantic and syntacticfeatures in the text according to a statistical model, wherein thestatistical model identifies the value associated with each feature inthe listing of features; and determining a text difficulty scoreassociated with the resource using each of the values assigned to theidentified semantic and syntactic features.
 15. The system according toclaim 14, further comprising assigning a weight to each of the assignedvalues; and determining a weighted value for each of the identifiedsyntactic and semantic features, wherein the text difficulty score isdetermined using the weighted values.
 16. The system according to claim15, further comprising outputting the text difficulty score to acomputing device of a content curator.
 17. The system according to claim16, further comprising adjusting the text difficulty score according toan input received from the computing device of the content curator. 18.The system according to claim 14, further comprising: grouping theresource with a set of one or more resources having a comparable textdifficulty score; and categorizing resources into one or more sets ofresources based on a difficulty category.